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SYSTEM AND METHOD FOR AUGMENTATION OF 
ENDOSCOPIC SURGERY 

BACKGROUND OF THE INVENTION 

1. Field of the Invention 

This invention relates to the field of endoscopic surgery. More 
specifically the invention relates to obtaining accurate positional 
information about an anatomical structure within a patient's body 
and using this information to accurately position endoscopic cam- 
eras and surgical instruments within the patient's body. 

2. Description of the Prior Art 

Systems have been developed to augment a human surgeon's ability 
to perform surgery on a patient by providing the surgeon with 
intraoperative images of anatomicalstructurcs within the patient's 
body. Typically, these systems comprise a specialized form of 
camera or medical telescope. Further, a class of these systems, 
which includes endoscopic and laparoscopic instruments, has re- 
duced Ihc invasive nature of many surgical procedures. 

This class of systems has two salient characteristics in common: 
First, the surgeon using the system cannot directly manipulate the 
patient's anatomy with his fingers, and second, the surgeon cannot 
directly observe what he is doing. Instead, the surgeon must rely 
on instruments that can be inserted through a trocar or through a 
working channel of an endoscope. Often, since his hands and at- 
tention are fully occupied in performing the procedure, the surgeon 
must rely on an assistant to point the endoscopic camera while the 
surgery is performed. 

To ameliorate the awkwardness of this arrangement, robotic aug- 
mentation devices have been developed for endoscopic surgery. One 
such device is described in detail in a copending application entitled 
"System and Method for Augmentation of Surgery" serial number 
07/ 714,816 f iled June 13, 1991 which is herein incorporated by ref- 
erence! ^ 

Robotic augmentation devices can potentially greatly assist sur- 
geons during an operation. Robotic devices do not fatigue. Poten- 
tially, they can position medical telescopes and surgical instruments 
very accurately and can perform precise repositioning and repetitive 



functions. However, in order for these advantages to be realized, a 
number of problems need to be solved. The surgeon still needs to 
determine what motions the robotic device is to make and requires 
a means to communicate with the computer controlling the robot. 
In a few cases, such as orthopaedic machining of bone or pre- 
planned excision of a tissue volume determined from preoperative 
medical images (such as CT Or MRI scans), these motions may be 
preplanned. However, in other cases, the surgeon needs to directly 
observe the patient's anatomy and interactively specify the motions 
to be made relative to anatomical features and the medical tele- 
scopes. In these cases, means of accurately locating anatomical 
features and instruments relative to the medical telescopes and to 
each other and of using this information to control the robotic aug- 
mentation aids are necessary. 

A specialized robotic device for stepping a rcscctoscope through a 
preprogrammed sequence of cuts in thranurcthral prostatectomies 
has been developed. However, this system docs not address the 
problem of providing the surgeon with a convenient means of con- 
trolling the view available through an endoscopic device or of pro- 
viding the surgeon with means of interactively manipulating 
surgical instruments in response to intraoperative imaging and other 
sensory information. 

There has been one attempt to provide voice control of a flexible 
endoscope in which servomotors attached directly to the control 
knobs of a commercial flexible endoscope were activated in response 
to voice commands by the surgeon. Difficulties of this approach 
include: (a) the surgeon (or an assistant) must still determine which 
direction to deflect the endoscope tip to provide a desired view and, 
consequently, must keep track of the relationship between the 
endoscope tip and the anatomical structures being observed; (b) 
these corrections must be made continually, distracting the surgeon 
from more important matters; and (c) the use of voice commands 
for this purpose is subject to arors, potentially distracting to the 
surgeon, and may make the use of voice for communication between 
the surgeon and operating room personnel more difficult. 

Several research efforts are directed to providing improved mech- 
anisms for flexible endoscopes. These devices do not, however, 
simplify the surgeon's problem of controlling the endoscopic camera 
to obtain a desired view, either by himself or by communicating 
with a skilled operator. 

3. Statement of problems with the prior art 
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Unfortunately, the medical telescopes which arc used in minimally 
invasive surgery have limited fields of view. As. a result, only a 
small part of the anatomical feature hidden inside the patient's 
^ body can be viewed at^one time. Furthermore, surgical telescopes 
typically provide only a single vantage point at any one time and it 
is difficult to provide the desired view. 

Normally, to compensate for this limited field of view, a surgical 
assistant operates thctelescope, reorienting it to produce many 
views of the anatomical feature. While doing this, the assistant 
must continuously keep track of the relative orientation between the 
telescope and the patient's anatomy in order to be able to quickly 
and correctly aim the telescope at the surgeon's request. He or she 
must also correctly interpret the surgeon's desires, which are not 
always evident from the surgeon's verbal comments. 

This creates a number of problems. Surgical procedures of this na- 
ture now require an additional highly-skilled person to assist the 
surgeon in manipulating the medical telescope because the surgeon 
is using both of his hands performing other tasks. The communi- 
cation that is required between (he surgeon and the assistant in- 
creases the potential for an error while performing the surgery. The 
surgeon (and assistant) have to develop and keep a mental image 
of the entire hidden anatomical feature because the telescope can 
not capture the full image of the feature. Many telescopes, whether 
flexible or rigid, provide an oblique view, i.e., the direction of view 
is not coincident with the main axis of the telescope. This further 
exacerbates the difficulties of correctly aiming the telescope to 
achieve a desired view and increases the likelihood that the surgeon 
or the assistant could misconstrue the image presented or lose the 
orientation of the telescope with respect to the anatomical feature. 
Human fatigue contributes to a degradation of positioning of the 
telescope and/or of the interpretation of the images that (he tele- 
scope transmits. 

Accordingly, there is a need for a way to obtain accurate and reli- 
able information about the position and appearance of anatomical 
features hidden within a body. There also is a need for an appara- 
tus to accurately position and orient surgical instruments and/or 
medical telescopes within a body and to provide accurate informa- 
tion about their position with respect to hidden anatomical features. 
Further, there is a need to provide a reliable and accurate interface 
between the surgeon and his surgical instruments so that he can 
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accurately position these instruments with respect to an anatomical 
feature within a body without removing his hands from his instru- 
ments. 



OBJECTIVES 

An objective of this invention is to provide an improved method to 
obtain and display accurate information about the position of an 
anatomical feature within a patient's body. 

Also an objective of this, invention is to provide an improved 
method of positioning endoscopic cameras and other surgical in- 
struments within a patient's body. 

A further objective of this invention is to provide an interface for a 
surgeon to accurately position an endoscopic camera and/or other 
surgical instruments within a patient's body without removing his 
hands from the instrument. 

BRIEF DESCRIPTION OF THE DRAWINGS 

Figure I shows a schematic view of a system used for computer 
augmentation of surgical procedures. 

Figure 2 is a detail of Figure 1 showing a distal fine motion rota- 
tional manipulator. 

Figure 3 shows an embodiment of the invention using a stereoscopic 
visualization system. 

Figure 4 shows an embodiment of the present invention comprising 
two robotic manipulators. 

Figure 5 shows positions in 2D and 3D Cartesian coordinate sys- 
tems. 

Figure 6 shows the pin-hole mathematical model of a camera. 

Figure 7 shows a method of computing a position in three dimen- 
sions using two nonparallcl camera vantage points. 
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Figure 8 shows the use of passive visual targets to determine a po- 
sition of a surgical instrument. 

Figure 9 shows a method of computing a position in three dimen- 
sions using two parallel camera vantage points. 

Figure 10 shows a method of using oblique medical telescopes. 



SUMMARY OF THE INVENTION 

The present invention is a method and apparatus for determining 
positional information about an object and then using this infor- 
mation to position instruments in relation to the object. The in- 
vention has many applications but is particularly useful when the 
object is hidden from view or in a location that is difficult to access. 
One preferred embodiment, used in endoscopic surgery, determines 
positional information about a designated anatomical feature which 
is hidden within a patient's body! The information is used to posi- 
tion surgical instruments in the body with respect to* the anatomical 
feature. 

The invention first positions an instrument, e.g. a surgical instru- 
ment inserted inside a patient's body, at a desired position relative 
to a designated object (anatomical feature). The instrument is ca- 
pable of transmitting an image of the object to a computer which 
then determines positional information about the object by using 
various types of image processing. The information is then related 
to a human (e.g., a surgeon) or to a computer controlling a robotic 
apparatus.- The positional information is used to position or repo- 
sition the transmitting instrument and/or other instruments relative 
to the designated object. 

To further facilitate use of the invention, <*i number of different 
output modes for conveying information from the imaging instru- 
ments and computer to humans in the operating room are provided. 

To further facilitate use of the invention, input devices are incorpo- 
rated on the inserted instruments so that a human user can input 
requests to the system while concurrently manipulating the instru- 
ment.. Other methods of inputting requests to the system, such as 
voice recognition systems, are also incorporated so that communi- 
cations with the system does not interfere with instrument manipu- 
lation. 
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DETAILED DESCRIPTION OF THE INVENTION 

Referring lo Figure 1, Ihere is shown a schematic view of a system 
for use in computer augmentation of laparoscopic or similar proce- 
dures. The system generally comprises a manipulator apparatus or 
robot 242, a computer 243, a drive motor interface 244, a 
monoscopic monitor 247 with a suitable image processor 245 and 
graphics adaptor 246, a stereoscopic monitor 272 with suitable 
stereo display system 271, and a terminal 248 for connecting addi- 
tional input devices to computer 243. 

A manipulator similar to the manipulator 242, used in this pre- 
ferred embodiment, is described in detail in the copending U. S/ 
application serial numbcr^07/7 J4,816 fi led on June 13, 1991. 

Referring to Figures 1 and 2, the manipulator 242 comprises a 
proximal rectilinear manipulator 6 and a remote center-of-motion 
distal manipulator 240. The proximal inanipulator 6 comprises 
three mutually orthogonal sliding motion sections 1, 2, and 3, which 
provide motion in the X, T, and Z directions. Sections 1, 2, and 3 
are equipped with computer-controlled motorized drives 4 con- 
nected to motion interface 244 and also have manual locking clamps 
5. The remote center-of-motion distal manipulator 240 comprises 
rotational sections 7, 250, 251, and 252 to provide 0 P Q x , 6 y , and 
distal 0 Z rotational motion, and a slide motor 253 adapted to axially 
slide instrument 254. These sections are equipped with computer- 
controlled motorized drives 249 interfaced to motor interface 244 
and have manual locking clamps 255. Each of the moving sections 
of manipulator 242 can be actuated cither manually or under com- 
puter control and can optionally be locked by a manual locking de- 
vice. All the motorized drives 4 and 249 arc controlled by computer 
243 through motor interface 244. 

Referring to Figure 2, there is shown a schematic view of the distal 
fine motion rotational manipulator, 240 with an instrument 241 in- 
serted through an incision into a patient's body. In the embodiment 
shown, the distal manipulator 240 provides a five degree-of-frecdom 
Ox, 0z, and d) remote center-of-motion wrist, which is sup- 
ported by the aforementioned proximal positioning system with 
three orthogonal linear degrees of freedom (X, K, and Z). The 
proximal linear degrees of freedom arc used to place the center-of- 
motion M of the remote center-of-motion wrist at the position of 
insertion into the patient's body P. Any alternative mechanical 
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structure (such as a SCARA manipulator, manufactured and sold 
by IBM) with sufficient degrees of freedom could be substituted for 
this purpose. 

The four distal revolute degrees of freedom and the sliding degree 
of freedom of manipulator 240 give the surgeon a five degree-of- 
freedom spherical work volume centered at the insertion point M. 
These degrees of freedom may be selectively locked or moved inde- 
pendently (manually or under computer control) to assist the sur- 
geon in achieving a desired precise alignment. Furthermore, small 
motions within the work volume can be achieved with only small 
motions of the individual axes. The point M (i.e., the point at 
which the surgical instrument enters the patient) remains unaffected 
by any motions of the distal manipulator 240. Thus the manipula- 
tor may be moved through its work volume without requiring that 
the patient position be moved or that the size of the entry wound 
be enlarged. 

One consequence of this design is that motion of the proximal ma- 
nipulator 6 is not needed unless the patient is moved. Conse- 
quently, in a preferred embodiment, the motion of proximal 
manipulator 6 is disabled by manual locking and/or disabling of 
drive motors whenever an instrument fs inserted into the patient. 
In this mode, the control computer 243 interprets commands re- 
questing motion of manipulator 242*as follows. When a motion is 
requested, the control computer 243 attempts to satisfy the request 
by moving only distal manipulator 240. If the motion can be ac- 
complished in more than one way, the computer selects the motion 
that minimizes the motion of the most proximal revolute motion 
section 7 (i.e., it minimizes motion of 0 P ). If the motion cannot be 
accomplished perfectly, the computer selects the motion of distal 
manipulator 240 that most closely approximates the desired motion. 
Modes are available to select minimization of positional error of the 
tip of instrument 24 1 ,' orientation crrots or weighted combinations 
thereof, if the error is greater than a prcspccificd threshold amount, 
the control computer notifies the surgeon using synthesized speech, 
an audible alarm, or other means, and makes no motion unless the 
surgeon explicitly instructs it to proceed, using voice recognition or 
other input modality. One alternative embodiment might seek al- 
ways to minimize the total motion of the distal manipulator 240, 
again forbidding motion of proximal manipulator 6 whenever a 
surgical instrument held by the distal manipulator is inserted into 
the patient's body. Yet another might permit small motions of the 
proximal manipulator, so long as the center-of-motion M stays 
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within a specified threshold distance (e.g., 3 mm) of the original 
value. 

If desired, a flexible tip may be added to the distal end of instru- 
ment 241 to provide additional degrees of freedom. In the case 
where a viewing instrument such as instrument 254 is used, an ad- 
ditional degree-of-freedom in adjusting the gaze direction may be 
provided by adding an adjustable-angle mirror or prism to the distal 
end of the instrument. 

Referring again to Figure 1, the instrument 254, in the embodiment 
shown, includes a video camera 259 and a light source 277 con- 
nected to the instrument via a fiberoptic cable 278. The video out- 
put of the camera 259 is fed into the graphics adaptor 246, where 
it may be optionally mixed with graphics output from computer 243 
and displayed on monitor 247. The video output from the camera 
is also optionally fed into the image processing system 245, which 
analyzes the image produced by the camera and provides informa- 
tion to computer 243 about the relative position of the surgeon's 
instruments, the camera, and the patient's anatomy. The video in- 
formation from the camera may be also optionally supplied to the 
stereo display system 271, which can assemble a stereoscopic view 
of the patient's anatomy from two or more images taken from dif- 
ferent vantage points and display the image on the stereoscopic 
monitor 272. 

In one preferred embodiment, the stereo display system is a 
StereoGraphics CrystalEyes (trademark of StcrcoGraphics, Inc.) 
system, where the two video signals are displayed on a stereoscopic 
monitor which alternatively displays the left and right eye image at 
a frequency of 120 Hz, updating the video information for each eye 
60 times per second. The surgeon wears stereoscopic liquid crystal 
(LC) goggles 273, which arc synchronized with the monitor and al- 
ternatively block light from entering left and right eye such that the 
left eye receives only the video signal from the left camera and the 
right eye receives only the information from the right camera. The 
frequency of alternation between left and right images is sufficiently 
high such that the surgeon perceives no flicker but rather a contin- 
uous stereoscopic image of the patient's anatomy. Other stereo 
display technologies arc available and may be used. 

In the embodiment shown, the surgeon is using a second surgical 
instrument 260 inside the patient's body, which has passive visual 
targets 276 placed on it. These targets 276 are markings on the in- 
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strument and are chosen so as to be easily locatable by the image 
processing system 245 in the images supplied by the camera 259. 

The set of input/output devices attached to input/output interface 
248 of computer 243 shown in Figure 1 may include a computer 
voice recognition and synthesis system 267, a joystick 268 mounted 
on the surgical instrument 260 and a sterilized touch screen 269 
mounted on monitor 247. In the preferred embodiment the joystick 
is a small device, functionally identical to a 2D or 3D mouse, but 
designed such that it can be mounted directly onto a surgical in- 
strument and such that at least two degrees of freedom of motion 
can be specified by applying pressure on a small joystick protruding 
from the device. One implementation of such a device uses strain 
gauges to translate an applied pressure or force into incremental 
displacement or velocity information. In another embodiment, a six 
degree-of-freedom input device, such as Spaceball (A Trademark 
owned by Spaceball Technologies, Inc.) could be used used to 
specify motion in any of the six degrees of freedom. Such a device 
could be mounted on a surgical instrument, on the manipulator 
structure, or at any other convenient point. One advantage of 
mounting an input device such as a small joystick on a surgical in- 
strument is that the surgeon can easily manipulate the joystick 
without removing his hands from the surgical instrument, thus per- 
mitting him to provide information to the computer (for example, 
of a desired direction of motion of a medical telescope) without in- 
terrupting his work. 

The speech recognition and synthesis system 267 includes means of 
inputting information to the system, such as a (possibly head 
mounted) microphone 275, and a means of conveying information 
to the surgeon, such as a speaker 274. The speech recognition sys- 
tem 267 is capable of understanding a vocabulary of instructions 
spoken by the surgeon and can relate the information about the 
commands it has received to the computer 243/ The surgeon may 
use any of these modalities, cither separately or in combination, to 
position graphic objects on the monitor 247, to select commands or 
operating modes from menus, and to command motions of the ma- 
nipulator 242. 

Referring to Figure 3, there is shown an alternative embodiment of 
the system for computer augmentation of laparoscopic or similar 
surgical procedures. In this embodiment, the surgical instrument 
254a is a stereoscopic medical camera, which incorporates two in- 
dependent lens systems or optical fibers and is capable of transmit- 
ting two simultaneous images from the paticnt/s body. The two 
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lenses are separated by a small (known) distance and are thus able 
to provide a stereoscopic image. One embodiment of such a device 
would comprise two side-by-side fiberoptic bundles or lens systems 
and one fiberoptic light channel. The assembly would be sur- 
rounded by a suitable cylindrical casing. The video signals from the 
two cameras 259a and 259b are fed into the stereo display system 
271 and displayed to the surgeon on a stereoscopic display monitor 
272. Using interface hardware known in the art, both video signals 
are also optionally supplied to the image processing system 245 and 
the graphics adapter 246. 

Another embodiment of the system is shown in Figure 4, where the 
system comprises two manipulators 240a and 240b, carrying surgi- 
cal instruments 241a and 241b, respectively. In one embodiment, 
one of the surgical instruments is a medical telescope, whereas the 
other instrument is a surgical tool, such as medical forceps. Since 
both instruments are attached to robotic devices, both can be ac- 
tively positioned under computer control. On the other hand, as 
with the single manipulator arm in the case above, either or both 
robots can be controlled manually by releasing, adjusting, and re- 
locking joint axes one at a time. In an alternative embodiment, both 
surgical instruments 24 1 a and 241b comprise medical telescopes or 
other means of transmitting an image outside of a patient's body. 
In such an embodiment, one of the instruments (for example, 241a) 
may also comprise a surgical tool such as a miniaturized surgical 
forceps. In this case, information from images taken at two vantage 
points may be combined to provide precise 3D information to assist 
in placing the surgical instrument precisely on the desired portion 
of the patient's anatomy. 

Referring again to Figure 1, the image processing system 245 may 
be used to locate features on the patient's anatomy of interest to the 
surgeon. In this mode, the surgeon would designate a feature of 
interest by any of a number of means to be explained below. On 
the surgeon's command, supplied via any appropriate input device 
attached to the terminal 248, the computer 243 would instruct the 
image processing system 245 to acquire an image and precisely lo- 
cate the designated anatomical feature. In one embodiment, a ref- 
erence image of the designated feature would be acquired in 
response to the surgeon's command and stored. Image correlation 
techniques would be used to locate the feature during surgery. In 
an alternative embodiment, synthetic reference images could be 
generated from computer reconstructions of preoperative medical 
images and models. Once a feature has been located, the manipu- 
lator 242 can be moved to place the feature at any desired position 
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in the camera field of view. If desired, an additional image may be 
acquired, the feature re-located, and a further adjustment made to 
refine the desired placement of the camera. This process may be 
repeated a number of times to "zero in" on a feature to any desired 
accuracy. Each of the foregoing steps is explained below. 

As a matter of nomenclature, we will in the following text refer to 
positional information in a number of ways. Unless otherwise 
specified, the terms "position" and "location" will be used inter- 
changeably. We will be referring to two-dimensional (2D) and 
three-dimensional (3D) positions. When referring to an image ob- 
tained by a single monoscopic camera, an "image location" or "im- 
age position" should be understood as a 2D location within the 2D 
image. Referring to Figure 5a, such a location A (within a 2D im- 
age 800) is given as a pair of coordinates (xy). When the image is 
stereoscopic, "image location" or "image position' 7 should be under- 
stood as a 3D location within the volume of the stereoscopic image. 
Referring to Figure 5b, such a location B is described by a triple of 
coordinates (xj>,z). We will also refer to positions of anatomical 
features. Such features are part of the patient's anatomy and all 
references to "feature location" or "feature position" should be un- 
derstood as 3D positional information about the feature in question. 

In order to use and manipulate images of the patient's anatomy, 
images must first be acquired. Referring to Figure 1, this is done 
by feeding the live video signal from camera 259 into the image 
processing system 245 comprising at least one video digitizer. A 
video digitizer is a device capable of converting an analog video 
signal into a digital signal, which can be stored in computer memory 
and arbitrarily modified by the computer. Conversely, a video 
digitizer can also convert a digitized (and possibly modified) video 
signal back into analog form for display on a standard monitor. 

If positional information is to be extracted from images obtained 
by a camera/lens system, a mathematical model of the camera and 
the lens must be available to relate image points (i.e., points on the 
camera's imaging plane) to the corresponding world points (i.e., 3D 
locations in the actual environment). To a good approximation, a 
perfect camera/lens system can be modeled as a pin-hole system, il- 
lustrated in Figure 6. The figure depicts a camera with a lens 600 
positioned a distance f in front of the image plane 601. The quan- 
tity / is referred to as the focal length of the lens. A point 
W= {xy,z) lying in the plane 602 a distance rf= - z in front of the 
lens is imaged onto the image plane 601 at the location C = (*',/), 
where xfd = x'jf and y/d = y'jf. 
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Given the image coordinates (x\y') of a world point, the above re- 
lationships constitute two equations in three unknowns (x, y, and 
z) and are thus not sufficient to recover the 3D coordinates of the 
corresponding world point, W. Referring to Figure 7, the informa- 
tion obtained from a single image 60 1 a from a first vantage point 
600a defines a ray 605a in 3D space originating at the image point 
C a , passing through the lens center 600a, and extending to infinity. 
By definition, the actual world point W lies somewhere on this ray, 
but additional information is needed to determine its exact location. 
If a second image 601b, taken from a second vantage point 600b 
(whose position and orientation with respect to the first vantage 
point is known), is available, then the corresponding image point 
C b in the second image and the location of the second vantage point 
600b define a second ray 605b in space, such that the world point 
W lies on this ray a well. Using known mathematical techniques, 
the two rays can be resolved in the same coordinate system and 
their intersection can be computed, giving the 3D world coordinates 
(jcj/,z) of the point W. 

Most camera lenses introduce distortions which causes the corre- 
spondence of world and image points to depart from the above 
pin-hole model. The process of calibrating the camera/lens system 
can estimate the nature and amount of such distortions and the re- 
sulting mathematical model can be used to effectively "undistort" 
the image points. The pin-hole camera model can then be applied 
to the undistorted image. A number of techniques for calibrating 
camera/lens systems are known. 

As part of the interaction with a two-dimensional image of the pa- 
tient's anatomy displayed to the surgeon on a conventional monitor, 
the surgeon may wish to designate (i.e., point to) a particular image 
location within the displayed image. The surgeon may point to a 
particular image location by using any of the following means: (a) 
by positioning a surgical instrument equipped with a distinct and 
clearly visible visual target so that the image of the visual target on 
the display coincides with the desired image location, (b) by ma- 
nipulating a graphical object on the screen using an input device 
mounted on a surgical instrument (such as joystick 268 in Figures 
1 and 3 or a similar device), or (c) by manipulating a graphical ob- 
ject on the screen using a conventional mouse. In method (a) the 
visual target may consist of a brightly colored spot or a known ge- 
ometric pattern of such spots at a known position on the instrument 
(e.g., pattern 276 in Figures 1 and 3). The use of a bright color, 
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distinct from any color naturally occurring inside the patient's body, 
greatly simplifies the problem of locating artificial visual targets and 
lessens the chances of erroneous location of such targets. Such spots 
on the surgical instrument can be located using known image proc- 
essing techniques, involving thresholding (to isolate the spots from 
the rest of the image) and computationally determining the centers 
of the so obtained thresholded regions. In methods (b) and (c) the 
position of the feature of interest is taken as the final position of the 
graphical object. 

Once the 2D coordinates of an image location have been specified 
to computer 243, the computer can confirm the location by marking 
the location with a graphical object superimposed on the image. In 
one embodiment of this method of confirming an image location to 
the surgeon, 2D cross-hair cursors or 2D box cursors can be used 
to show the location of interest in the image. The "image", in this 
context, can be either a TV camera image or a computer generated 
graphical rendition of the anatomical area of interest. 

We have so far described a variety of methods for the surgeon to 
specify a particular 2D location of interest in a monoscopic image. 
We next discuss methods, such as image processing, to determine 
positional information about three-dimensional anatomical features 
and/or surgical instruments in the patient's body. 

Referring to Figures I and 3, if a stereoscopic display (live or static) 
of the patient's anatomy is available during the surgical procedure, 
then a surgeon can designate the desired 3D anatomical feature of 
interest by manipulating a 3D stereoscopic graphical object (cursor) 
on the stereoscopic display 272 until the graphical object is coinci- 
dent with the desired anatomical feature. Any of the appropriate 
aforementioned input devices and modalities 248 (such as the sur- 
gical tool mounted joystick or trackball, voice, etc.) can be used to 
specify the desired motion of the graphical object within the 
stereoscopic volume of the image. 

If the actual physical size of a designated object is known, its dis- 
tance from the viewing instrument may be estimated from the size 
of its image, as seen by the viewing instrument. Since we know that 
the feature lies on a ray originating at the center of image of the 
feature and passing through the vantage point as shown in Figure 
7, the position of the feature relative to the viewing instrument may 
then be computed. Let the size of the feature in the image be /, let 
the corresponding actual size of the feature be .v, and let / denote the 
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focal length of the camera. The distance z from the camera lens to 
the feature of interest can then be computed as z = {fx -0//. 

Referring to Figure 8, in one embodiment, where passive visual 
targets 701 on a surgical instrument 700 are used, the position of a 
3D feature (e.g., a surgical instrument 700) can be determined as 
follows: At least three non collinear circular spots 701 of known di- 
ameter s are marked on the surgical instrument 700 (Figure 8a). 
Since the surgical instrument may have an arbitrary orientation 
with respect to the camera, these spots will in general appear on the 
image plane as ellipses 705 (Figure 8b). However, the length of the 
major axis of each ellipse / will be the same as the diameter of the 
circular image that would be seen if the corresponding circular spot 
were presented at that same distance from the lens in such a man- 
ner that the plane in which it lies is perpendicular to the view axis 
of the camera. Let the length of the major axis of the observed el- 
lipse as it appears in the image be / (Figure 8b). Then the distance 
of the spot from the camera lens can be computed from 
z~ (fx s)jL Having performed this computation for at least three 
spots and knowing the position of the spot pattern with respect to 
the tip of the surgical instrument suffices to compute the 3D lo- 
cation of the tip of the surgical instrument with respect to the cam- 
era. Other techniques, known in the art, permit calculation of the 
position and orientation, relative to the camera, of a pattern of five 
dots from the 2D positions of their centroids in the image obtained. 
Other patterns of dots or other visual targets can be used as well. 
The 3D location of the tip of the instrument relative to the camera 
may then be readily computed from the known position of the tip 
relative to the visual target. 

Additionally, stereo image processing may be used to precisely lo- 
cate 3D anatomical features. In one embodiment, image processing 
can be used in conjunction with a stereoscopic camera to locate an 
anatomical feature. Referring to Figure 3, surgical instrument 254a 
is a stereoscopic medical camera, comprising of two independent 
lens systems or optical fibers and is capable of transmitting two si- 
multaneous images from the patient's body. The lenses arc sepa- 
rated by a small (known) distance d, as shown in Figure 9. The 3D 
position of the anatomical feature relative to the camera tip can be 
computed from the pin-hole camera model (Figure 6). Specifically, 
if the image plane locations of the center of the feature of interest 
in the two images are denoted by f { = (x h v*i) and f 2 = (x 2 ,y2), as 
shown in Figure 9, then the distance z of the feature center from the 
came ra lens can be computed as z = if x d)jc, where 
c = J(x 2 — x y ) 2 + (y 2 — y\) 2 and / denotes the focal length of the 



YO9-92-080 



-14- 



# 



camera. Image correlation techniques or other image processing 
techniques known to the art may be used to locate features in im- 
ages. 

Referring again to Figure 9, in another embodiment, using only a 
monocular camera, image processing techniques can be used to de- 
termine the position of an anatomical feature in three dimensions 
as follows: A first image 601a of the anatomical feature is acquired 
and a reference representation (such as a multi-resolution image 
pyramid representation known in the image processing art) is 
stored. The manipulator 242 is used to displace the camera lens tip 
600a laterally by a known amount d, and a second image 601b is 
acquired. The center of the feature of interest W is located in the 
second image, using the reference representation of the feature, by 
means of correlation techniques (such as multi-resolution normal- 
ized correlation methods known in the art) and the 3D displacement 
of the anatomical feature from the camera tip may be computed as 
in the case above. Specifically, if the image plane locations of the 
feature of interest W in the two images 601a and 601b are denoted 
by /, = (x u y x ) and f 2 = (x 2 ,j; 2 ), respectively, then the distance z of 
the feature from the camera lens can be computed as z ~ {f x d)fc, 
where c = J(x 2 — x x ) 2 4- (y 2 - j'i) 2 and / denotes the focal length of 
the camera. 

In another embodiment, the physical constraint of maintaining 
minimal translational motion of the telescope with respect to the 
port of entry into the patient's body may preclude laterally dis- 
placing the telescope to obtain a second image, as described above. 
Referring to Figure 7, in this embodiment, a first image 60 1 a is ob- 
tained from the first vantage point and the center W of the feature 
of interest is located in the image at image location C a The telescope 
is then rotated by a small (known) amount about the port of entry, 
such that the desired feature is still within the field of view of the 
telescope, and a second image 601b is obtained. Note that the sec- 
ond vantage point has a different position and orientation than the 
first vantage point. The feature center VV is located in the second 
image at image location C b . The 3D position of the feature center 
W is then obtained by computing the intersection of the rays 605a 
and 605b, as described previously. As above, image correlation 
techniques or other image processing techniques known to the art 
may be used to locate features in images. Alternatively, the surgeon 
may be asked to manually designate the image location of the fea- 
ture center in the two images using any of the means of designating 
image locations described previously. 
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Once a 3D feature has been designated and its 3D location suc- 
cessfully computed, computer 243 can confirm its location by 
marking the location with a 3D stereoscopic graphical object 
superimposed on the stereoscopic image of the area of interest. In 
one embodiment of this method of confirming 3D feature location 
to the surgeon, 3D cross-hair cursors or 3D box cursors can be used 
to show the feature's 3D location within the stereoscopic view vol- 
ume. The "image", in this context, can be either a TV camera im- 
age or a computer generated graphical rendition of the anatomical 
area of interest. 

Once the 3D positions of anatomical features are stored in com- 
puter 243, this information may be used to control the position and 
orientation of the camera tip relative to the features so as to provide 
any desired field of view. 

Referring to Figure 1, in one mode the surgeon can designate a first 
and a second 2D location in the image, using any of the means Re- 
designating 2D locations discussed above. The surgeon can then 
instruct the manipulator 242 (using any appropriate input device 
or modality as described earlier) to reposition the camera tip 266 so 
that the anatomical feature f h whose image appeared at the first 2D 
image location prior to camera motion, appears at the second 2D 
location in the image after the camera motion. The distance of the 
camera tip 266 from the anatomical feature f x remains constant 
during the camera motion. A special case of this mode is the case 
where the second 2D location is the center of the image. In this case 
the camera is repositioned so that the anatomical feature f { appears 
to move to the center of the displayed image, i.e., the camera is 
"centered" over the anatomical feature. 

In another mode the surgeon can specify a sequence of 2D locations 
in an image and instruct the manipulator 242 to move the camera 
tip 266, at a constant elevation, so that the camera traverses the 
path defined by the sequence of 2D locations in the image. In one 
embodiment, this sequence of image locations can correspond to 
image locations of distinct small anatomical features within the 
camera's field of view. In another embodiment, the sequence of 
image locations can correspond to image locations of a boundary 
of a large anatomical feature, such as a blood vessel. This mode of 
repositioning the camera can be viewed also as specifying the de- 
sired apparent motion of an anatomical feature (corresponding to 
the last 2D location in the specified sequence) with respect to the 
image. The term "apparent motion of an anatomical feature" is 
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used to emphasize that the anatomical feature does not physically 
move, but only appears to move relative to the image due to the 
motion of the camera. Specifically, the execution of this mode pro- 
ceeds as follows: The sequence of 2D image locations is processed 
by computer 243 into a continuous path by the process of interpo- 
lation. The camera is then centered over the anatomical feature 
corresponding to the first designated 2D image location as described 
in the previous paragraph. The camera is then repeatedly posi- 
tioned so as to center each of the successive interpolated 2D lo- 
cations within its field of view, thereby effectively traversing the 
path as defined by the surgeon. The surgeon directly controls both 
the direction and speed of the camera motion by means of the sur- 
gical tool mounted joystick or any other appropriate input means. 

In another mode the surgeon can specify an increment of motion 
along the camera's axis of view and reposition the camera along this 
axis by the designated amount. The "axis of view" in this context 
is defined as the line joining the camera lens center and the point p 
on the patient's anatomy which appears in the center of the camera 
image. This mode effectively implements a zoom function with re- 
spect to a 3D anatomical feature, where the zoom factor (i.e., de- 
sired enlargement or contraction of the image of the anatomical 
feature) is specified by the surgeon interactively. In particular, this 
mode can be implemented by allowing the surgeon to interactively 
manipulate a graphical cursor on the screen whereby he can specify 
the desired zoom factor by enlarging or contracting one such cursor 
with respect to a reference cursor whose size does not change during 
the zoom factor specification. Any appropriate input device 248 
can be used to manipulate the cursor object. Computer 243 then 
uses the relative geometry of the two cursors to compute the direc- 
tion and magnitude of the camera motion increment, which is nec- 
essary to effect the specified zoom factor. Alternatively, voice input 
can be used to specify the zoom factor. Once the camera motion 
increment has been computed, computer 243 instructs manipulator 
242 to (slowly) reposition the camera Up 266 by that amount along 
the axis of view, thereby obtaining the desired zoom factor. Note 
that the point as defined above, remains at the center of the im- 
age throughout the zooming process. 

In another mode, the surgeon can directly control a desired direc- 
tion of motion of the camera vantage point via an instrument- 
mounted input device. In the preferred embodiment, this input 
device is a six dcgrce-of-frccdom joystick. Using such a joystick, the 
surgeon can then arbitrarily reposition and reorient the camera in 
all six degrees of freedom simultaneously. By selecting different 
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subsets of the full six degree-of-freedom motion, a number of useful 
control modes can be implemented. In particular, if the 
translational controls of the six degree-of-freedom joystick are disa- 
bled or only a three degree-of-freedom input device is available, a 
camera motion control mode can be implemented, where the camera 
tip is constrained to move along the surface of an imaginary sphere, 
centered at the current anatomical feature of interest and having 
radius equal to the current distance of the camera tip from the fea- 
ture. In another embodiment, where only a two degree-of-freedom 
input device is available, any two of the six degrees of freedom can 
be controlled by the device at any given time. For instance, pressing 
a two degree-of-freedom joystick in the direction toward the tip of 
the instrument on which the joystick is mounted can be interpreted 
to mean "zoom in", and pressing away from the tip can mean 
"zoom out". Releasing the joystick can mean "stop". Similarly, 
exerting pressure or force on a two degrce-of-freedom joystick in a 
direction perpendicular to the long axis of the camera can be inter- 
preted by computer 243 to mean a desired lateral motion of the 
camera at the current elevation in the direction of the exerted pres- 
sure. Additionally, the velocity of the camera motion can be made 
proportional to the amount of exerted pressure on the joystick. 

In another mode the surgeon can manipulate a graphical object 
superimposed on the image of the patient's anatomy to specify a 
desired view of a particular feature of interest. The camera is then 
automatically positioned to achieve the desired view. A particular 
implementation of this mode would proceed as follows: An image 
of the patient's anatomy is obtained and displayed to the surgeon 
on a display monitor. The surgeon is then allowed to designate a 
feature of interest in a 2D or 3D image, unless the desired feature 
has already been designated and is visible. Next the surgeon can 
interactively manipulate a graphical object (e.g., cursor, slider, etc.) 
superimposed on the image of the patient's anatomy on the display 
screen to specify the desired view of the feature of interest. For 
example, the view specification could specify the desired vantage 
point of the camera anywhere on the surface of a sphere of a given 
radius centered at the feature of interest. Computer 243 then com- 
putes the appropriate displacement of the camera and instructs the 
manipulator 242 to execute the motion, thereby obtaining the de- 
sired view of the feature of interest. 

If the surgical augmentation system comprises two independently 
controlled robotic systems, as illustrated in Figure 4, another mode 
of using the 3D positional information about the patient's anatom- 




YO9-92-080 



-18- 



i 

ical features to reposition a surgical instrument can be used, where 
the instrument being repositioned is the second surgical instrument, 
rather than the surgical telescope. In one embodiment of this in- 
vention, the second surgical instrument could be surgical forceps, 
which are repositioned such that the jaws of the instrument are co- 
incident with the current 3D anatomical feature and a tissue sample 
of this feature can thus be obtained by closing the instrument's 
jaws. 

Referring to Figure 10, the capability of interactively designating 
the desired view of a particular 3D feature of interest and letting the 
computer compute the resulting new location of the medical tele- 
scope is especially important in situations where the telescope's op- 
tics provide a lateral, rather than a straight-ahead (a — 0°) view. 
Telescopes with the direction-of-vicw anywhere between 30° and 
135° (with respect to the instrument's long axis) are commonly used 
in laparoscopic and similar procedures. Figure 10 illustrates a tele- 
scope with the direction-of-vicw of a = 45°. Manually positioning 
such a telescope to achieve a desired view can be extremely difficult 
even for an experienced camera operator as the relative transfor- 
mations between the telescope, the patient's anatomy and the image 
coordinates become rather complex and unintuitive. However, 
adding a single rigid body transformation to the computational 
chain in the computer software accounts for the fact that the 
dircction-of-view is different from 0°. In a particular implementa- 
tion, a coordinate frame F c is associated with a 0° telescope, and the 
computer keeps track of the rigid body transformations between the 
manipulator, the camera, and the various anatomical features of 
interest. The mathematical methods and techniques of representing 
and manipulating rigid body transformations are well known to the 
art of robotics and computer graphics. Camera motions needed to 
effect a particular zoom factor, for example, arc then computed 
relative to this camera frame F c . For the case of a non-straight 
telescope, such as the telescope in Figure 10, a new coordinate 
frame F c - is defined by rotating the frame F c through an angle of 
— a about a line passing through the center of the lens tip and par- 
allel to the A'-axis of the coordinate frame F c . The rigid body 
transformation c T c: relating the new camera frame F c - to the default, 
0° location of the camera frame F c , is used to account for the non- 
zero direction of view of the telescope. Using the transform F c: in 
place of F c in the computation of the new desired position of the 
telescope for a particular desired view now results in correct reposi- 
tioning of the telescope regardless of its dircction-of-view. 
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The visual information transmitted from the patient's body and 
optionally augmented by image processing and computer graphics 
can be displayed to a surgeon in a number of ways. 

Referring to Figure 1, in one mode of information display, the im- 
ages of the patient's anatomy can be displayed to the surgeon as a 
combination of live and still images (a live image is an image ob- 
tained from the camera that is continuously updated with new in- 
formation, whereas a still image is not). In one embodiment of this 
mode, the image to be displayed on the monoscopic monitor 247 is 
produced as follows: A wide-angle monoscopic image of the pa- 
tient's anatomy is obtained using the surgical instrument 254 and 
displayed on the monitor 247 as a static image. The camera is then 
zoomed in for a closer view of the current feature of interest and a 
portion of this live TV image is displayed superimposed on top of 
the static wide-angle image. The static monoscopic view of the 
overall area of interest thus provides contextual information about 
the patient's anatomy under observation, whereas the live subimage 
shows a magnified detail area surrounding the current anatomical 
feature of interest. 

In an alternative embodiment of this display mode, the static wide- 
angle contextual information can be a computer-graphic rendering 
of the patient's anatomy. This graphical information can be derived 
from computer models of the patient's anatomy constructed on the 
basis of the information gathered during preoperative imaging and 
scanning. As before, a portion of the image surrounding the current 
anatomical feature of interest is replaced with a live magnified TV 
image of this area. Here, the computer generated image and actual 
live TV image are merged into a single display image and must thus 
be properly registered with respect to each other to ensure proper 
correspondences of anatomical points and features between the two 
images. A number of techniques for achieving registration between 
images arc known to the art. In the simplest embodiment, the 3D 
locations of a number of known anatomical landmarks represented 
in the computer model would be identified by 3D image processing 
techniques. The 3D locations of these landmarks can then be used 
to compute the appropriate perspective view for displaying the 
graphical model. 

In another embodiment of this display mode, the static wide-angle 
contextual information can be a computer-graphic rendering of the 
patient's anatomy, as above. Similarly, a portion of the image sur- 
rounding the current anatomical feature of interest is replaced with 
a live magnified TV image of this area, as above. In addition, the 
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live TV image of the area of detail can be augmented by superim- 
posing static edge information, which can be derived either from a 
computer graphics model or as a result of image processing (edge 
extraction) on the TV image. The advantage of this display mode 
is that the superimposed edges highlight the ongoing changes within 
the area of detail reflected in the live TV image with respect to the 
previous (static) appearance of this area. 

In another embodiment of this mode of displaying information to 
the surgeon, the static wide-angle view of the overall area of interest 
can be displayed as a static stereoscopic image. Referring to Figure 
1, this is achieved as follows: A static image of the overall area of 
interest is obtained from a first vantage point using the surgical in- 
strument 254 and camera 259. The tip of the camera lens 266 is 
then displaced by a small known amount and a second static image 
of the area of interest is taken from this displaced vantage point 
The two images are then fed as input to the stereo display system 
271 and displayed on the steroscopic monitor 272 as a static 
stereoscopic wide-angle view of the overall anatomical area of in- 
terest. In some cases where only the distal manipulator 240 is 
moved to displace the camera, there may be some small angular 
misalignment of the two images so obtained. Experiment has shown 
that this misalignment can often be ignored, since the human visual 
system is very adept at fusing slightly misaligned images. Alterna- 
tively, the misalignment can be largely compensated for by using 
image transformation techniques known in the art. Next, the cam- 
era is zoomed in for a close-up view of the current anatomical fea- 
ture of interest and a portion of the static wide-angle image is 
replaced by the magnified live monoscopic view of the anatomical 
feature of interest, as before. This results in an image, where the 
overall contextual information is a static stereoscopic image, pro- 
viding the surgeon with a sense of the global three-dimensional re- 
lationships within the viewing volume, and the area surrounding the 
current anatomical feature of interest, where the surgeon's concen- 
tration is focused, is magnified and displayed as a live monoscopic 
image. 

In a modification of the above mode of display, the live TV image 
of the area of detail can be augmented by superimposing static edge 
information, which can be derived cither from, a computer graphics 
model or as a result of image processing (edge extraction) on the 
TV image. As described previously, the advantage of this display 
mode is that the superimposed edges highlight the ongoing changes 
within the area of detail reflected in the live TV image with respect 
to the previous (static) appearance of this area. 
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Referring to Figure 3, another embodiment of the present invention 
regarding display of visual information to the surgeon, uses the 
stereoscopic medical camera 254a to obtain a static stereoscopic 
wide-angle image of the overall anatomical area of interest. Then, 
as above, the stereoscopic camera is zoomed in closer over the cur- 
rent 3D anatomical feature of interest and a portion of the static 
image surrounding the feature of interest is replaced by a magnified 
live stereoscopic TV image as transmitted from the patient's body 
by cameras 259a and 259b. 

In order to emphasize the changes occurring within the area of de- 
tail, edge information corresponding to a previous state of the area 
of detail can be superimposed on the live stereoscopic image, as be- 
fore. 

Again referring to Figure 3, another embodiment of the present in- 
vention uses the stereoscopic medical camera 254a in conjunction 
with stereoscopic computer graphics to provide a display of the pa- 
tient's anatomy. In this embodiment, the static stereoscopic view 
of the overall anatomical area of interest is derived from computer 
models of the patient's anatomy and displayed on the monitor 272 
as a 3D stereoscopic graphical image via the stereo display system 
271. As above, the stereoscopic camera is then zoomed in closer 
over the current 3D anatomical feature of interest and a portion of 
the static graphical image surrounding the feature of interest is re- 
placed by a magnified live stereoscopic TV image as transmitted 
from the patient's body by cameras 259a and 259b. 

Again, in order to emphasize the changes occurring within the area 
of detail, edge information corresponding to a previous state of the 
area of detail can be superimposed on the live stereoscopic image, 
as before. 

Referring to Figure 1, another mode of display of anatomical in- 
formation to a surgeon uses the monoscopic camera 254 to provide 
the surgeon with a live stereoscopic image of the patient's anatomy. 
In this mode, the information supplied to one of the surgeon's eyes 
is derived from computer models of the patient's anatomy and is 
displayed as a graphical image computed from the vantage point 
displaced a small known distance laterally from the current vantage 
point of the surgical instrument 254. The information supplied to 
the other eye is the live image of the patient's anatomy as provided 
by the camera 259 attached to the surgical instrument 254. In this 
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mode, one eye therefore receives static computer generated view of 
the patient's body, whereas the other eye receives a live image 
transmitted by the camera from a slightly displaced vantage point. 
If the computer-graphic model is properly registered with the actual 
anatomy, the human brain will fuse the two images into a proper 
3D stereoscopic image. 

In another embodiment of the above mode of display of anatomical 
information to the surgeon, image processing is used in conjunction 
with live video information to produce a live stereoscopic display to 
the surgeon. Referring to Figure 1, in this embodiment of the 
present invention, a first image of the patient's anatomy under ob- 
servation is obtained and transferred to the image processing system 
245. The camera tip 266 is then displaced laterally a small known 
amount and a second image is obtained from this second vantage 
point and transferred to the image processing system 245. The im- 
age processing system and known image processing techniques are 
then used to extract edge information from the two images. A 
stereoscopic display is then produced by supplying the stereo dis- 
play sj'stem 271 with only edge information in one of the input 
channels (left/right eye) and a live video signal with overlaid edge 
information in the other input channel (right/left eye). Subse- 
quently, only information to one of the two eyes is updated with live 
video as transmitted by camera 259. This provides enough infor- 
mation for the human brain to Till in" the missing information and 
interpret the image as a proper stereoscopic 3D image. 

Alternatively, a display mode as above can be used, where the edge 
information is not obtained by image processing, but rather derived 
from a computer graphical model of the patient's anatomy. 

Aside from visual information, the surgeon can receive non-visual 
information about the locations of features or the general state of 
the system as well. One non-visual channel of communication be- 
tween the surgeon and the system is the voice recognition and 
speech synthesis subsystem (267, Figure 1). For example, synthe- 
sized voice messages can be issued by the system to inform the sur- 
geon of the exact location oT his surgical instrument with respect to 
an anatomical feature of interest. Likewise, synthesized messages 
confirming successful receipt of a voice command can be used to 
assure the surgeon that the system correctly interpreted his 
command(s). General system state or change of system state infor- 
mation can be relayed to the surgeon using synthesized voice as 
well. An example of this would be a synthesized speech message to 
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the surgeon stating the exact distance by which the camera was 
moved during a zooming operation. 

An alternative method of relaying non-visual information to the 
surgeon is tactile feedback. In one embodiment of this invention, 
tactile feedback conveyed to the surgeon through a hand-held or 
instrument-mounted input device (such as a joystick) can be used 
to alert the surgeon that he has positioned a graphical object or a 
surgical instrument in the vicinity of the current anatomical feature 
of interest. The tactile feedback can be delivered to the surgeon's 
hand or finger (whichever is in contact with the joystick) by instru- 
menting the joystick control with a computer controlled vibrator. 
When the vibrator is activated by the computer, the joystick control 
starts vibrating with appropriate frequency and amplitude, such 
that the oscillations are readily discernible by the surgeon, but do 
not distract him from his positioning task or otherwise interfere 
with his work. 
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